DOCUMENT RESUME 



ED 228 284 



AUTHOR 



TITLE 



INSTITUTION 

SPONS AGENCY 
REPORT NO 
PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



TM 830 171 

Raupp, Magdala / 

Considerations in Decijding on a District Management 
Strategy for Linking Testing and Evaluation Practices 
with Instructional Change. ^ 
Calif ornia Univ. , Los Angeles. Center for the Study, 
of Evaluation. 

National Inst, of Education (ED), Washington, DC. 
CSE-R-183 

82 - 
15p. 

Reports - Descriptive (141) ' 
MF01/PC01 Plus Postage. 

Educational Change; "Educational Strategies; 
Elementary Secondary Education; "Evaluation 
Utilization; Formative Evaluation; "Instructional 
Improvement; "Management Systems; "School Districts; 
Test Use; Test Validity 
"Linkage 



ABSTRACT 

The process of developing a 
testing/evaluation/instruction management subsystem that will be 
uniquely suited to the specific situation of a given district 
requires a decision to use a strategy for instructional change iln 
which data from testing and evaluation would £lay a major role. I 
Considerations that might go into such a decision involve some of the 
elements of an analytic framework specified by Bank and Williams , 
(1981). Major questions are:Ml) whether a 

testing/evaluation/instruction management strategy is worth the 
effort; (2) what opportunities and constraints are posed by the 
relevant environments and how will these environments shdpe the 
strategy; (3) what ideas related to testing and evaluation run 
consistently through the district; and (4) what effect would these 
ideas have on strategy choice and implementation. Further 
considerations are the operational elements necessary to implement a 
management subsystem, the formal and informal structures that will 
serve as coordinating mechanisms to make testing and evaluation 
useful to teachers, and the impact of the subsystem on instruction. 
(CM) 
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The intent of the Evaluation Design Project conducted at, the 
Center for the Study of Evaluation; with Adrianne Bank and Richard 
C. Williams as co-directors, was to examine a small sub-set of existing 
district policies, procedures, and programs that appear to contribute 
to systematic-use of evaluation ftnd testing for educational improvement. 
It was expected that the project, by addressing itself to the influences 
on arid the concerns of school and district personnel, would be in a 
position to^make suggestions for other districts' guidance. 

Through' an extensive nomination procedure, six districts were iden- 
tified whose administrators claimed their district was attempting to 
use test scores or evaluation data as a guide to revising one. or more 
aspects of their instructional activities. During subsequent interviews 
with district and school personnel in each of the districts, we found a 
variety of practices that districts used to link testing and evaluation 
with instruction. We found that the specific practices differed from dis 
trict to district.as did the districts' intraorganizational structures 
and the sequence in which their* management subsystem linking testing and 
evaluation evolved. 

In the districts we^visited, we saw practices that can be roughly 
grouped into three configurations. These configurations will be referred 
* to as/strategfes and are fully described in the paper School District 
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Management Strategics to Link Testing with Instructional Change t (Bank, 
1981). We*do not mean that these three are the only possible strategies nor 
that the districts s&t out to create a system to implement a particular 
strategy. , 

1 . An instructiona11y-oi\iented, objectives-based strategy . 4 
This district adopted \tructuretl diagnostic/prescriptive 

* teaching suppoA^d by a\ district-wide scope and sequence 
outline of objectives, a\cri terion-rieferenced testing system ^ 
continuously updated, and Materials cross-referenced to the - V 
objectives and to the test? 

2. A personnel-oriented staff development strategy . 
Great school -to-schgol variability and, the likelihood that 

principal and teaching staffs wduld remain, stable influenced 

f » * ( ■ 

at least one of the districts to adopt a personnel -oriented staff 

development strategy as a key to data based instructional change. 

Their assumption was that teachers themselves made the major 

difference in student learning and.* that data about def-i cits 

in student achievement^«l»uld determine the content of staff 

development courses. 

A building-oriented problem solving strategy . Schools in this 
district, for reasons of ethnicity, geography, and tradition, re- 
presented distinctive organizational entities. The district 
felt that school staffs and parents should together identify 
their problemscind deyise solutions using testing and evaluation 
data.* * 



Although the term strategy will bring to mind direction and 
purpose, we found that most districts we visited had not'made a plan 
or a blueprint prior to taking the actions that seemed to result. in a 
strategy. What they did evolved out of events, interests of people, . 
effects of the environment.. At some point in time, these activities . 
were conceptualized or reconceptualized so that future activities 
could be justified or made plausible.* At that point, and- for some dis- 
tricts, what could be Called *a strategy emerged. 

A wide range of reasons influenced pur six districts to do what 
they did to -use data to influence instructional decisions. Clearly the 
immediacy of state or federal mandates and funds was one factor. Obvious 
shortcomings in student, ski 11 &, in some' cases, internal pressures within 
particular district offices, or the special interests of . individuals in 
positions of power on the Board or within the district provided the im- 
petus. Districts deemed to be, in this manner, adjusting and accommo- 
dating - in the Piagetean sense of the phrase - to- the^various" inputs and 
demands made upon^them. 

Althpugh what we described above was the pattern we most often 
fourfd, we suggest that it impossible for leaders in districts to visualize 
and shape a management strategy in a way that is more proactive for 
subsequent activities. It- should be possible for leaders to understand 
many of the constraints and influences active on and within the tli strict 
and then take reasonable steps to move in an instructional change direction. 
'If we believe public schools should be seeking instructional change thereby 
improving student learning, and if we believe testing and evaluation can 



contribute to this change, we should clarify, as best we can,- the 
process 4>f developing a testing/evaluation/instruction management sub- 
system that will be uniquely suited to the specific situation of a 
given district. , 

In this paper we will. deal with some,. of the considerations that might 
go intb a decision to use a strategy for instructional change in which 
data from testing and evaluation wtuld play a major roleV We will u$e, 
as organizers, ^soife of the elements of an analytic framework specified 
by Bank and Williams (1981). • There are certain things these elements^ 
can do and other. things that they cannot do ? They can help us think 

about "the district as ah organization embedded in and responsive to "its 

■ i 

environment. They can bring attention to ideas, operations, and mechanisms* 



already in place. An analysis using these elements can make the decision- 
making process somewhaf less uncertain. Such analysis \wfll raise but 
not answer our questions. The decision to develop a management strategy 
and the specifics of that strategy must be unique to a district and not 
acquired as a shelf-item from other districts. While it is true that many 

i districts share common characteristics, in no two districts is the £fom- .^y 
bi nation and arrangement of characteristics the^same. . 

Commitment to develop a strategy for managing data based instructional 
change will lead district personnel into a multitude of decisions and 
considerations of complex. issues. Since effort beyond that required^ 
simply to maintain the status^quo is needed, a first major question 
might be: IS SUCH A T/E/I MANAGEMENT STRATEGY WORTH THE EFFORT? The 

* answer to that general' question may be forthcoming from answers to more 
specific questions. 
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are administrators, teachers, and parents satisfied with what the 

districts' schools are doing in relation to student learning? 

do, these individuals see a ga| between wh^t^he schools are 

doing and what it would bet possible do? \ * m 

do these individuals believe that educational leadership requires 

activism and a constant search for better alternatives? 
i ■* . • . ' 

8 do these" individuals believe that more effective instruction 

can come, at least in- part, as a result of examining the pre- ^ 
sent performance of students on tests or'th^ present performance 
of programs^as indicated by evaluations? - ( 
A second question, concerned with relevant environments, 
might be: WHAT ARE THE OPPORTUNITIES AND THE CONSTRAINTS PO&ED BY THE 
ENVIRONMENT IN WHICH THE DISTRICT ORGANIZATION IS EMBED0§D? . HOW WILL. 
THESE ENVIRONMENTS SHAPE THE MANAGEMENT STRATEGY? 

Relevant environments include factors external to the district or- 
ganization but which act upon it.' Geographic factors such as the exis- 
tence of schools isolated or clustered together^ community factors, such 
as size, its socio-economic, political, or religions organization; his-^ 
toric factors such as' its past tradition or reputation and its expecta- 
tions for the" future. Such factors in the environment can help or hinder 
data-based Jpstructional change. Answers to ^ht^sipecific questions below 
might help to answer the more 'general foregoing question. 

8 how have testing and evaluation results been historically used 
ir^this district? 
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,! do parents and members of the district community usually share 
the belief that" test results reflect the learning of students 
and that the district has responsibility for incasing that • 
learning through instruction? . . 
* what specific instances are there of -this belief? Is it shared 

by all parents? ^ t : 

V what are the external pressures and incentives to use testing and 

' i - 

evaluation results to improve instruction? have there been federal 
or court .mandates? how strong a pressure has the media, . through, 
its critical appraisal of the public school system, puiTon the 
^district? , . * ■ 

» what is" the attitude of the opinion leaders irrthe community to- 
wards the iss.ue~of more efficiency in the schools and how suppor- 
tive would they be in the -event that the district commits itself 
to a strategy for improvement? 
0 geographically, what J s the organization of the schools with- 
in the district? are the same issues being considered at the 
dif-fl#ent schc/ols or are the issues and problems different at 
each school site? 

A third general question might be related to ideas , an elemen£ in 
e framework that refers to ideas specific to testing, evaluation, and -in 
struction and how- they interact with one another; also included' would 
be those ideas specific to the management of the subsystem linking test- 
Wand evaluation with instruction: WHAT IDEAS RELATED TO. TESTING AND 
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VALUATION RUN CONSISTENTLY THROUGH THE DISTRICT AND HOW WOULD THEY AFFECT 
CHOICE AND IMPLEMENTATION OF A STRATEGY FOR INSTRUCTIONAL IMPROVEMENT? 
Are there ideas that converge to a common vision valued by members 
of the staff, members of the community, and parents? Group\members have 
images of themselves and agree on the way in'which they perceive the 
world, based on common experience, values, and beliefs. Future^ action- 
is going- to be based on those ideas that members of groups share. It 
1* important,' therefore, to probe in^the content and direction pf^these ' 

ideas. \ 

° what, ideas do the teaching staff have about testing and evaluation 
that would support'or undermine the effort to use data generated by 
^both for instructional .improvement? ^ ^ 

8 what "proportion of the staff would exper^TCeanxiety or threat , 
regarding the use of test results to diagnose^ deficits in the 
instructional process? , ...... 

8 what proportion of the staff would be. willing to undergb training in- 
new skills and behaviors necessary to establish the test, evaluation, 

and 'instruction 'link? 

\ 

8 what' proportion of individuals ^.members of the community, parents, 
media people <t would be willing to participate in the effort? 

°-< is' there* a. 'critical mass' or total number of individuals within 
the staff, members of the ^mmunity, and parents that are genuinely 
concerned,over. the issue of instructional improvement and would 
be supportive of me'asures taken in that direction? t > : 

° does this 'critical mass! see the proposed action as appropriate 
and adequate, that it will move the schools in the direction of the 
target, and that the benefits will outweigh' the costs? 



to what extent are thos« In leadership gosi tigns. able to create 
a common vision ttaat is valued by many members and a sense of 
urgency, of needing to take initiative and, action? 
° are tjier'te 'idea champions ' . that could imfrue the, staff with improve- 
ment fervor? what is the personal' style of. those individuals, 
what makes them important' and how could they become an ^asset to 

the strategy under consideration? . y % 

» 

Another major question could be posed in relation to those operational 
elements that are to be included in the management subsystem, .such as, 
testing, evaluation, instruction, curriculum, supervision, staff development, 
budget,' personnel, media: WHAT IS THE EXTENT AND KIND OF EXPERT FUNCTIONS 
" NECESSARY* TO SUCCESSFULLY IMPLEMENT A DATA-BASEQ INSTRUCTIONAL IMPROVEMENT 
PROGRAM? * * 

° \rowg^uch and whatjiind of technical knowledge is available with- 
in districl^nit/s or at the school site? how can those resources 
be tapped? 

if kj^wledge^md.-e^e^^e outside the district have to be utilized, 
where is i t^n^QWr^ffcessibX is it? 

what factors ^^nongly influence teachers' decisions about the content" 

of instruction? how do these factors relate to the findings from 

■ - $ « ■ ■ 
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testing programs? 

/ " 1 ' 

how might staff development activities deal with testing and evaluation, 

especially with their linkages to instruction? who would be pre- c 

pared to conduct training 1 in the development of appropriate tests, 

interpretation of the information they 'generate, and the use of 

the information for diagnostic' and prescriptive purposes? 



V what. effects does the current testing prqgraif have on the curriculum 
and how dt these effects -differ from the effects expected if a - 

x - ' i ■/ & • 

m linkage system were established? how migjit this linkSge shape 
the curriculum? : ^ 

° what financial resources are available in the form of money, 
( services, or Materials and what guidelines and procedures are re- 
quired for their use? what. strings are attached ta the funds? 
what kind or supporffrom the media -san be counted ypon and how < 
would this support help the effort? , 
The element of coordinating -mechani'sms refers to both formal and 
informal structures that function to maximize staff commitment to, and sta- 
communication -about data-6ased instructional change, "Our major question 
then would be; 1 HOW CAN THE ORGANIZED UNfTSA^D LEVELS* WITHIN THE prSTRlCT 
AND THE ROLES- OF DIFFERENT ACTORS BE CONNECTED SO AS TO PLACE TESTING AND 
EVALUATION ACTIVITIES II>.A POSITION TO ASSIST TEACHERS? More specific f% 
questions might include*. . ' . 

° how.might the information generated by testing and evaluation reach 

* 

the different groups andn'u what form? who woifld make use of- 
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this information? 

4 t t 

is there complexity "in the organization of the district such that' 
additional demands foade, by the change effort would be difficult 
tcT^cope with? would problems such as scheduling, limited personnel, 
staff turnover reduce the likelihood of carrying the change through? 
° what has the testing and evaluation branch of the 'district a Iread* 
<^dpne that has truly influenced instruction? how do .the evaluation 
staff now relate to the staff of other branches and'to the teaching 



and admini strati ve^taffs at each school site? does the evaluation 
branch staff have the skill to work effectively with diverse views? 
opinions, and values? > 
° do different groups within the district communicate their meanings 
and intentions clearly, use appropriate decision-making methods, 
and involve a wide range of appropriate persons in the decision- 
making process? 

The last element in the framework, impact , includes the ways in 
which the management subsystem might affect instruction, whether that 
effect is intended or unintended. The major questions then become: 
WHAT ARE THE DESIRED- EFFECTS OF LINKING TESTING- AND EVALUATION PROCEDURES 
TO THE INSTRUCTIONAL ACTIVITIES CONDUCTED AT THE SCHOOL? WHICH GROUPS 
DO. WE HOPE TO IMPACT? WHAT MIGHT CONSTITUTE OBSERVABLE OUTCOMES? WHAT 
HARMFUL CONSEQUENCES SHOULD WE TRY TO ANTICIPATE AND AVOID? Specific 
questions might include: 

° what instructional improvements would be considered satisf&tory 
in relation to the effort made? how would this instructional im- 
provement be measured? are there short and long range goals to 
be attained? 

° which groups might experience impact as a result of the management 
strategy? how would impact differ from group to group? what 
would each group be expected to do as a result? ■ • 
°how would ideas and attitudes related to testing and evaluation 
change as a result of the intervention? 
Summary ,- The purpose of this paper was to suggest to educators content . 
plating a strategy for data based instructional change questions that might 
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assist their thinking about the process. • Data generated by tests 
and evaluation may be able to provide a sound bisis for the management 
of the instructional system but such use requires a complex series of 
technologies and understandings. 
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